Method and Apparatus for Automated Plant Necrosis

ABSTRACT

A method of real-time plant selection and removal from a plant field including capturing a first image of a first section of the plant field, segmenting the first image into regions indicative of individual plants within the first section, selecting the optimal plants for retention from the first image based on the first image and the previously thinned plant field sections, sending instructions to the plant removal mechanism for removal of the plants corresponding to the unselected regions of the first image from the second section before the machine passes the unselected regions, and repeating the aforementioned steps for a second section of the plant field adjacent the first section in the direction of machine travel.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/720,021 filed on Dec. 19, 2019 (now U.S. Pat. No. ______, issued on______), which is a continuation of U.S. patent application Ser. No.15/665,025 filed on Jul. 31, 2017 (now U.S. Pat. No. 10,524,402, issuedon Jan. 7, 2020), which is a continuation of U.S. patent applicationSer. No. 14/713,362, filed on May 15, 2015 (now U.S. Pat. No. 9,756,771,issued on Sep. 12, 2017), which is a continuation of U.S. patentapplication Ser. No. 13/788,359, filed on Mar. 7, 2013 (now U.S. Pat.No. 9,064,173, issued on Jun. 23, 2015), which claims the benefit ofU.S. Provisional Application Nos. 61/608,005 filed Mar. 7, 2012 and61/609,767 filed Mar. 12, 2012, all of which are herein incorporated intheir entirety by reference.

TECHNICAL FIELD

This invention relates generally to the agricultural field, and morespecifically to a new and useful method and apparatus for automatedplant necrosis inducement.

BACKGROUND

Induced plant necrosis, such as crop thinning, is a common practice inagriculture, in which plants are selectively removed from densely seededplant beds to provide the remaining plants with adequate space forgrowth. Conventional crop thinning is performed manually, wherein aworker walks along a crop row and removes plants within the crop rowwith a hoe at his discretion. Not only are these methods costly and timeconsuming due to the use of human labor, but these methods also fail tooffer a maximization of plant yield over the entire field, as the workertypically focuses on a single row and does not select plants forretention based on inter-row packing. While automatic crop thinningsystems exist, these systems fail to offer the plant removal flexibilityin plant selection and removal that human labor offers. In one example,a conventional crop thinning system removes plants at fixed intervals,whether or not the plant removal was necessary. In another example, aconventional crop thinning system removes plants using system vision,but fails to identify multiple close-packed plants as individual plantsand treats the close-packed plants as a single plant.

Therefore, there is a need in the agriculture implement field for a newand useful method and apparatus for automated inducement of plantnecrosis.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a schematic representation of the method of automatedinducement of plant necrosis.

FIGS. 2A, 2B, and 2C are schematic representations of a variation ofsegmenting the foreground from the background within an image or fieldof view, identifying points of interest within the image, andclassifying points of interest as plant centers within the image usingmachine learning, respectively.

FIGS. 3A, 3B, and 3C are schematic representations of a second variationof classifying the points of interest as plant centers within the image,including identifying points of interest in a first image, identifyingpoints of interest in a second image, and classifying recurring pointsof interest between the two images as plant centers, respectively.

FIGS. 4A and 4B are schematic representations of presenting the imageand identified plant centers to a user and reassigning the points ofinterest as plant centers based on the user input, respectively.

FIG. 5 is a schematic representation of segmenting the image intoregions and sub-regions representative of plants.

FIG. 6 is a schematic representation of capturing an image of a crop rowsegment.

FIGS. 7A and 7B are schematic representations of updating a virtual mapwith the regions and sub-regions associated with plants from a first andsecond image, respectively.

FIG. 8 is a schematic representation of the crop thinning mechanism.

FIG. 9 is a side view of the detection mechanism utilized with a plantbed.

FIG. 10 is a side view of a variation of the crop thinning mechanismincluding a first and a second detection mechanism and a first and asecond elimination mechanism.

FIG. 11 is a schematic representation of a crop row.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following description of the preferred embodiments of the inventionis not intended to limit the invention to these preferred embodiments,but rather to enable any person skilled in the art to make and use thisinvention.

1. The Plant Necrosis Inducement Method

As shown in FIG. 1, the method of automated plant necrosis includescapturing an image of a plant field section, identifying individualplants within the image S100, selecting plants for retention from theimage S200, removing plants from the plant field section S300, andrepeating the aforementioned steps for a following plant field sectionS400. The plants removed by the method preferably include crops, but canalternatively include weeds or any other suitable plant. Likewise, theplant field sections are preferably crop rows, but can alternatively beweed rows, weed sections, or any other suitable portion of an areacontaining plants. Plant removal preferably includes inducing plantnecrosis, but can alternatively include eliminating the plant or anyother suitable method of killing given plants. The method is preferablyperformed by a system including a detection mechanism and a eliminationmechanism. This method affords several benefits over conventionalsystems. By automating the plant removal process, this method allows forfaster plant removal over that of manual methods. Furthermore,automation allows for optimization across the entire field of theretained plants for space, size, density, health, or any other suitableparameter. Automation also allows for quality control by removing thesubjectivity of the human that was thinning the plants and improving theconsistency of the retained plants. By identifying individual plants,this method allows for individual plant targeting for removal orretention, making the crop thinning process more reliable and the cropthinning results more predictable over that of conventional systems.

The field in which the method is used preferably includes a plurality ofparallel crop rows including the same type of crop (e.g. same genus,same species, etc.). Alternatively, the field can include a first and asecond crop, both of which are to be thinned. Alternatively, the fieldcan include one or more main crops and a plurality of secondary plants,wherein the secondary plants are the plants to be removed. The crop rowsare preferably spaced between 2 inches to 45 inches apart (e.g. asdetermined from the longitudinal row axis), but can alternatively bespaced any suitable distance apart. The crop is preferably lettuce, butcan alternatively be corn, soy beans, carrots, tomatoes, broccoli,cabbage, or any other suitable commercial crop. The plant canalternatively be a weed or any other plant matter to be removed.

Identifying individual plants S100 functions to distinguish individualplants within an image of contiguous, close-growing plants. Morespecifically, identifying individual plants functions to determine anidentifying feature that is preferably only exhibited once in eachplant. The identifying features are preferably plant centers, but canalternatively be stalks, points of contact with the ground, or any othersuitable identifying feature. Determination of the identifying featuresallow for more reliable removal and/or retention during the plantremoval step. This is distinguished over conventional systems, whichtreat the contiguous plants (e.g. plants in contact with each other orpartially occluding each other) as a single plant due the contiguousplants' appearance as a substantially continuous region (e.g., a single“blob”). As shown in FIGS. 2 to 5, identifying individual plantsincludes identifying a foreground region within the image S110,identifying points of interest within the foreground region S120,classifying the points of interest as plant centers or non-plant centersS130, and segmenting the foreground region into sub-regions S140,wherein each sub-region encompasses a single point of interestclassified as a plant center. The image from which the individual plantscan be a frame of a video or a static image. Each image can be analyzedindependently (diachronically) and/or in comparison to other images toaccount for changes in time (e.g. synchronically).

In addition to identifying individual plants, the plant bed area, theplant bed longitudinal axis, the distance between adjacent plant beds(e.g. rows), row locations, parallelism, and coherence (e.g. how wellplants are aligned within a given row) or any other suitable informationabout the plant beds and plant field can be estimated based on sensordata. This information can be used to improve the computationalperformance of point of interest identification. For example, the croprow can be identified within the image using the Hough transform,wherein the row parameters (or points of interest) can be firstidentified at a certain linear and/or angular location within theresultant Hough space. However, rows can alternatively be detected usinga linear least-squares fit of point of interests along a row, using aleast-squares method with a regression approach (e.g. outlier detectionthrough maximum likelihood estimation or a RANSAC method), using aspecially-calibrated clustering method of selecting inliers, (e.g. aclustering method that considers elliptically shaped neighborhoods ofvarious points, etc.) can be used to recursively add points of interestto define a crop row. The recursion considers the neighborhood of pointsthat are already considered part of the cluster from a prior iteration.The larger major axis (in the row-wise direction) and the smaller minoraxis (normal to the row) allows for the inclusion of points of interestthat are closely associated with a row but may be farther away from eachother in the row-wise direction while rejecting outlier points ofinterest that may be closer to other point of interest but are too farfrom the row. In effect, this point-clustering method emphasizes therow-wise direction in the selection of points rather than give equalweight for points in all directions (as in the circular neighborhoodapproach). The image can be initially filtered using image-basedfiltering (e.g. adaptive binarization, color analysis, etc.) orvideo-based filtering (e.g. applying a Kalmann filter) to reduce noiseand sensitivity to changing ambient light conditions.

As shown in FIG. 2A, identifying a foreground region within an imageS110 includes segmenting the image into a background and a foreground.The background is preferably associated with the ground (e.g. soil ordirt), but can alternatively be the sky or any other suitablebackground. The foreground is preferably associated with one or moreplants. Foreground segmentation is preferably performed in real-time,but can alternatively be performed at any suitable frequency. The imageis preferably binarized to segment the foreground from the background,but can be otherwise processed to segment the foreground from thebackground. The binarization is preferably used as a mask such that onlypixels that correspond to the foreground are analyzed. The foregroundcan be segmented based on differences in depth (e.g. wherein theforeground is determined to be closer to the viewing plane),colorfulness, chroma, brightness, or any other suitable image parameteror measured parameter. The foreground can be detected using a setthreshold, a temporal average filter, running a Gaussian average,running a Gaussian mixture model classifier, or through any othersuitable foreground detection or segmentation method. In one variation,the plant-pixels are converted to the HSV color-space so thatconspicuous saturation in green is indicative of the presence of plantmaterial. In another variation, the image can be filtered for excessivered colorization and excessive green colorization.

As shown in FIG. 2B, identifying points of interest within theforeground region S120 functions to identify potential plant centerswithin each foreground region. The plant center is preferably the regionof the plant from which the upper plant leaves extend, as seen in atop-down view, and is preferably representative of the stem, theinternode, the apical bud, the leaf nodes of the upper leaves, or anyother suitable central portion of the plant. The plant edge ispreferably the region of the plant surrounding the plant center, and ispreferably defined by the blades of the plant leaves. Each extractedpoint of interest preferably includes one or more features that a plantcenter is expected to exhibit. The points of interest can be darkregions surrounded by one or more colors associated with a plant (e.g.particularly when low incidence lighting is used to capture the image).The points of interest can alternatively be regions within the imagehaving shapes (e.g. borders) similar to an apical bud, shapes similar toa shadow cast by the topmost leaves on the plant center, shapes similarto the leaves along the plant body, colors similar to an apical bud orcentral flower, or be any other region having a parameter similar tothat which the plant center is expected to exhibit.

The points of interest can be identified through saturation valuesadjacent the point of interest, through feature detection gradientanalysis (e.g. wherein points of interest include regions in theforeground surrounded by sharp gradient changes or high levels ofcurvature in the image gradient), saturation analysis (e.g. whereinpoints of interest include highly saturated regions in the foreground),hue analysis, brightness analysis, shape analysis (e.g. through anapplication of shape filters or edge detection), blob analysis, acombination of the above, or through any other suitable method ofdetermining a point of interest within an image or video stream. When apoint of interest is identified, the position of the point of interestwithin the image is preferably recorded along with the defining featuresof the point of interest (e.g. gradient curvature, saturation, area,shape, etc.). Points of interest are preferably extracted from everyforeground region identified in the image, but can alternatively beextracted from only the foreground regions indicative of multiplecontiguous plants (e.g. wherein the foreground region is larger than apredetermined size threshold, has a perimeter similar to an empiricallydetermined profile of multiple contiguous plants, etc.), whereinforeground regions indicative of a single plant (e.g. wherein the sizeof a foreground region is within a predetermined range, etc.) arepreferably considered a single plant. In the latter case, the plantcenter for the foreground regions indicative of a single plant can bedetermined as the centroid of the region, a point equidistant from alledges of the region, a point randomly selected within the region, or anyother suitable point within the region.

Classifying the points of interest as plant centers or non-plant centersS130 functions to determine individual plants within the image and toindirectly determine the location of the plant center within a crop row.Classifying the points of interest can include categorizing the pointsof interest into a plant center group or a non-plant group, can includeassigning weights or confidence levels to each point of interest,indicative of the likelihood that said point of interest is a plantcenter, or can include any other suitable method or combination thereofof classifying the points of interest. While all points of interestwithin a given foreground region are preferably classified, a subset ofpoints of interest can alternatively be classified, particularly whenthe non-classified points of interest satisfy exclusion parameters (e.g.the distance between the point of interest and the nearest edge is lowerthan a given percentage of the distance between the point of interestand the furthest edge, the point of interest area is below a sizethreshold, etc.).

In one variation, as shown in FIG. 2C, the points of interest can beclassified using machine learning algorithms or artificial intelligence,wherein the machine learning or artificial intelligence algorithms arepreferably supervised learning algorithms trained on a labeled set ofexamples (e.g. images of plants with pre-identified plant centers) butcan alternatively be unsupervised learning, semi-supervised learning,reinforcement learning, transduction, or utilize any other suitablemachine learning or artificial intelligence algorithm. The machinelearning algorithms preferably classify the points of interest based onfeature descriptors, such as binary robust independent elementary(BRIEF) descriptors, Histogram of Oriented Gradients (HOG) features,color histograms, or any other suitable features. The machine learningalgorithms can include Support Vector Machine (SVM), Naive Bayes, or anyother suitable algorithm. Alternatively, machine learning algorithms canbe used to determine the confidence level for each point of interest,wherein the confidence level is the classification confidence.

In another variation, as shown in FIG. 3, each point of interestclassified as a plant center is preferably assigned a default confidencelevel S132 (as shown in FIG. 3A), which can be subsequently adjusted.The confidence level for the point of interest is preferably increasedif the point of interest is identified or extracted from a predeterminedarea or pixel-neighborhood (after accounting for movement of the system)in a subsequent image S134 (as shown in FIG. 3B), and preferablydecreased otherwise. The confidence level for each point of interest ispreferably updated with each new image, wherein the decision whetherretain or remove the plant associated with the point of interest ispreferably made prior to the point of interest exiting the images.Alternatively, the confidence level of the point of interest can beproportional to the inverse of the square of the distance to the nearestpoint of interest to the estimated position of the first point ofinterest in the subsequent frame, after accounting for movement of thesystem. However, the confidence level for the point of interest can beotherwise determined. The point of interest is preferably classified asa plant center when the associated confidence level exceeds apredetermined threshold S136 (as shown in FIG. 3C), but can be otherwiseclassified. However, any other suitable method of automaticallycategorizing the points of interest can be used.

As shown in FIG. 4, classifying the points of interest S130 canadditionally include displaying the points of interest classified asplant centers S138 (as shown in FIG. 4A), receiving input from a user,and reclassifying points of interest as plant centers and non-plantcenters S139 (as shown in FIG. 4B). Displaying the points of interestclassified as plant centers S138 preferably includes generating andsending instructions for display of the points of interest classified asplant centers to a display device (e.g. a laptop, tablet or smartphone).The points of interest classified as plant centers are preferablydisplayed with an indicator (e.g. a red dot, a colored areacorresponding to the area of the point of interest, etc.) overlaid onthe image. Receiving input from a user preferably includes receivingfeedback from the user regarding the accuracy of the classification.Receiving input from a user can include receiving a plant center input(e.g. the user indicates the position of a plant center on the image), aplant center removal input (e.g. the user indicates that a point ofinterest classified as a plant center is not a plant center), a plantcenter repositioning input (e.g. the user moves the position of theplant center on the image), or any other suitable user input.Reclassifying points of interest as plant centers and non-plant centersS139 preferably includes reclassifying the points of interest accordingto the received user input, wherein regions of the image indicated to beplant centers are reclassified as plant centers and regions of the imageindicated to be non-plant centers are reclassified as non-plant centers.Reclassifying points of interest can additionally include adding theuser-edited image to the training set for the machine learningalgorithms, wherein the reclassified plant centers can be used to betterrefine plant center identification, and the reclassified non-plantcenters can be used to better refine non-plant center classification.This step is preferably performed in near real-time, preferably beforeplant removal instruction generation.

Segmenting the foreground region into sub-regions S140 functions toidentify the image area associated with each individual plant and toindirectly identify the area occupied by the respective plant within thecrop row. Each substantially continuous foreground region that includesmultiple points of interest classified as a plant centers is preferablysegmented into the respective number of sub-regions, wherein eachsub-region preferably encompasses a single point of interest classifiedas a plant center and can include any suitable number of points ofinterest classified as non-plant centers. Continuous foreground regionsencapsulating a single point of interest classified as a plant centerare preferably left unsegmented and are considered individual plants,but can alternatively be segmented to better delineate the area occupiedby the plant. A substantially continuous foreground region including afirst and second point of interest classified as plant centers can besegmented into a first and second sub-region by identifying acenterpoint equidistant from the first and second points and identifyinga centerline equidistant from the first and second points, wherein thecenterline intersects the centerpoint and a first and second opposingpoint on the foreground region edge. The first sub-region is preferablydefined by the foreground region edges proximal the first point ofinterest and distal the second point of interest and the centerline,wherein the first sub-region is preferably entirely encapsulated by theforeground region edges and centerline. Likewise, the second sub-regionis preferably defined by the foreground region edges proximal the secondpoint of interest and distal the first point of interest and thecenterline, wherein the second sub-region is preferably entirelyencapsulated by the foreground region edges and centerline.Alternatively, as shown in FIG. 5, each sub-region can be defined by arectangle or any other suitable polygon or shape. When a polygon isused, the centerline preferably intersects the polygon at a corner, andthe polygon preferably intersects at least one point on the foregroundregion edge proximal the respective point of interest, more preferablyat least a first, second, and third point on the foreground region edgeproximal the respective point of interest, wherein the first edge pointpreferably opposes the second edge point across the respective point ofinterest, and the third edge point preferably opposes a point on thecenterline across the respective point of interest. When the sub-regionsare defined by ovals, the ovals preferably intersect the centerpoint andas many points on the foreground edge proximal the respective point ofinterest as possible. The portions of the sub-regions can overlap,wherein the overlapping region can be accounted for in both the firstand the second sub-regions. However, the foreground region can besegmented into sub-regions in any other suitable manner.

Identifying individual plants can additionally include capturing animage of the plant field section S150, as shown in FIG. 6. Morepreferably, an image of a crop row section is captured, but an image ofa section of any other suitable plant field can alternatively becaptured. The image preferably includes a first crop row section and asecond crop row section, wherein the first crop row section issubstantially parallel to and adjacent the second crop row section.However, the image can be a section of a singular crop row, three ormore crop row sections, or capture a representation of any suitableportion of the plant field. Capturing an image of the crop rowpreferably includes radiating plants within a crop row with an emitterat an angle selected to cast a shadow at a shared plant feature (e.g.the plant center) and to radiate a portion of the plant surrounding theplant feature, and collecting the reflected radiation off the plantswith a detector. More preferably, capturing an image of the crop rowincludes illuminating plants within the crop row with a light directedat an angle between the ground and a normal vector to the ground, andcollecting an image of the illuminated plants with a camera. However,the emitter can be arranged to illuminate a desired plant feature (e.g.the plant center, near the stem) or to differentiate between any othersuitable plant features. The detector is preferably oriented such thatthe view plane is substantially parallel to the ground, but canalternatively be oriented in any other suitable angle. The plants arepreferably illuminated with visible light, more preferably white lightincluding cyan light (e.g. such as that produced from a white LED and acyan LED), but can alternatively include infrared, near-infrared,ultraviolet, or any suitable radiation. The wavelengths and intensity ofthe radiation produced by the emitter are preferably substantiallyconstant, but can alternatively be variable. In one variation, theprocessor adjusts the emitted wavelengths to supplement the radiationprovided by the environment (e.g. sunlight) when certain wavelengths aredetected to be below a predetermined threshold, increases the intensitywhen an image parameter (e.g. contrast) needs to be adjusted asdetermined through the confidence levels of the image analysis, orotherwise adjusts the radiation emitted by the emitter. In anothervariation, the processor modulates each of a plurality of lights betweena high and low state in a pre-set pattern, pseudo-random pattern, or anyother suitable pattern to facilitate point of interest identification.

Identifying individual plants can additionally include creating avirtual map of plants S160, which functions to map the sub-regions orregions indicative of plants that are extracted from the image to avirtual map of the plant field. As shown in FIG. 7A, the virtual mapfunctions to identify the relative position of each plant within thefield, and can also function to identify the relative position of thecrop thinning system relative to each identified plant. The virtual mappreferably correlates directly with the actual positions of the plants(e.g. is a 1:1 representation of the analyzed portion of the plantfield), but can alternatively be any other suitable representation ofthe actual plants on the field. As shown in FIG. 7B, the virtual map ispreferably dynamically updated S162 and/or expanded with each successiveimage. Each sub-region or region encompassing a single point of interestclassified as a plant center is preferably treated as a plant, and ispreferably mapped to a position within the virtual map. The positions ofthe plant centers are preferably represented on the virtual map. Thesub-region or region features, such as size, shape, color, or any othersuitable feature, can additionally be represented on the virtual map.The positions of the plant centers are preferably identified from theposition of the detection mechanism when the image was captured, and theposition of the points of interest classified as plant centers withinthe image. The position of the detection mechanism is preferablydirectly tracked or indirectly tracked (e.g. wherein another point ofthe crop thinning system that is statically fixed to the detectionmechanism is tracked) by a position-determining mechanism. Theposition-determining mechanism can determine the absolute geographiclocation, such as a global positioning system, a cellular towertriangulation system, or any other suitable location device.Alternatively, the position-determining mechanism can facilitaterelative distance determination from an initial point (e.g. a startingpoint), such as an accelerometer, a wheel encoder, a gyroscope, or anyother suitable acceleration velocity, or distance measurement mechanism.Alternatively, any suitable combination of measurement mechanisms can beused. Alternatively, the position-determining mechanism can utilizefilters or any other suitable means of determining the position of thedetection mechanism or crop thinning system. For example, tracking thecrop thinning system position can include finding correspondencesbetween points of interest in a current and subsequent frame, removingoutliers by the Random Sample Consensus (RANSAC) method, and performingan iterative closest point (ICP) calculation to arrive at rotational andtranslational matrices that describe how points are projected from aprior frame to a subsequent frame.

Selecting plants for retention S200 functions to determine the plants toretain and the plants to remove from the field segment represented bythe image, given only current and historical data (e.g. the currentimage, selections of plants for retention in previous sections, and datacollected from prior crop thinning procedures). Selecting plants forretention preferably includes selecting sub-regions or regions from theimage, wherein the selected sub-regions or regions are preferablyindicative of individual plants determined for retention, but canconversely be indicative of the plants determined for removal. Selectingplants for retention in real-time can additionally include mapping thesub-regions to actual plants within the crop row and tracking a plantlocation relative to a point on the crop thinning system.

Selecting sub-regions from the image preferably includes determining anoptimal pattern of retained plants that best meets a set of cultivationparameters for the section of the plant field represented by the image,based on the parameters of each sub-region, and selecting thesub-regions within the image that correspond to the retained plants toobtain the optimal retained plant pattern. The cultivation parameterscan include the intra-row distance between adjacent plants (e.g.distance between plants in the same row), inter-row distance betweenadjacent plants (e.g. distance between plants in different rows), yield(e.g. number of plants per unit area or plant density), uniformity inplant size (e.g. uniformity between the retained sub-regions and/orforeground region area), uniformity in plant shape (e.g. uniformity inretained sub-region and/or foreground region perimeter), uniformity inplant appearance, plant size and/or shape similarity to a given sizeand/or shape, the confidence or probability that the sub-region orregion is a plant, the practicality of keeping the respective plant,uniformity or conformance of measured plant health indicators to a planthealth indicator threshold, or any other suitable parameter that affectsthe plant yield that can be determined from the information extractedfrom the image.

Determining the optimal retained plant pattern preferably includescalculating a utility or cost value for each of a set of permutations ofretained plants, and selecting the optimal (e.g. maximum utility orminimum cost) permutation. However, the optimal retained plant patterncan be otherwise determined, such as through categorization,classification, optimization through randomized selection ofsub-regions, or in any other suitable manner. The set of permutations ofretained plants for which the utility or cost values are calculatedpreferably accounts for the retained (or conversely, removed) plants incrop row sections that have previously been thinned, such that the setof considered permutations is limited to those permutations includingtheoretically retained plants that correspond to actual retained plantswithin the respective crop row section, and theoretically empty spaces(e.g. no plants) that correspond to actual empty spaces within therespective crop row section. This is preferably accomplished by takingthe permutation that is selected for a first section as a given in theoptimization of a subsequent section, wherein permutations for thesecond section are determined by concatenating possible retained plantpermutations for the second section onto the selected permutation of thefirst section. Selection of the optimal permutation includes calculatingthe utility or cost function for the concatenated permutations, andselecting the optimal concatenated permutation. However, the constrainedset of permutations can be otherwise determined, such as by identifyingall possible permutations for the current and all previously encounteredsections, regardless of which plants were actually removed, thenremoving the permutations that do not match the actual pattern from theset of considered permutations.

Optimizing the cost or value function for intra-row and inter-rowdistance preferably maximizes plant packing while maintaining a selectdistance between the plants. For example, when the plant is a head oflettuce, the selected pattern preferably maintains a 8 inch or moreradius about each plant. The optimal pattern is preferably determined byoptimizing for Kepler's sphere packing problem, but can alternatively bedetermined in any other suitable manner. Optimizing the cost or valuefunction for yield preferably best meets a predetermined plant densityfor the analyzed section or for all considered sections, includingpreviously thinned sections and the currently analyzed section.Optimizing the cost or value function for yield can additionally includeselecting the plants that maximize the area for each plant, wherein theestimated area for each plant is determined from the corresponding“blob” or area occupied by the corresponding sub-region within theimage. Optimizing the cost or value function for uniformity in plantsize, shape, or appearance can include selecting the pattern withretained plants having the most similar size, shape, or appearance tothose plants retained in previous sections. Optimizing the cost or valuefunction for confidence preferably maximizes the confidence that thesub-regions and regions selected for retention are indeed plants. Thiscan be particularly useful when classification of the points of interestas plant centers and non-plant centers is probabilistic. Optimizing thecost or value function for practicality preferably accounts for thepracticality of keeping each plant, wherein a plant otherwise selectedfor retention can be marked for removal if the cost of removal is lessthan the value of retention. In a first variation, a plant otherwiseselected for retention can be marked for removal if the plant is in tooclose proximity to another plant marked for removal (e.g. the plantcenters are too close, the plant overlap is over a predeterminedthreshold as determined from the margin of error or precision of theelimination mechanism, etc.). This is preferably determined bydetermining the amount of overlap or interference between the two plants(e.g. as determined from the corresponding regions in the image),wherein the plant previously marked for retention is preferably selectedfor retention if the amount of interference falls below a predeterminedinterference threshold. Determining the amount of interference caninclude determining the distance between the plant centers, wherein theplant previously marked for retention is preferably selected forretention if the distance is above a predetermined distance (e.g. asdetermined from the elimination mechanism precision, such that the plantto be removed is far enough away from the plant to be retained that theelimination mechanism has a low probability of accidentally removing ordamaging the plant to be retained as well). In another variation, aplant otherwise selected for retention can be marked for removal if theelimination mechanism removal pattern cannot retain the plant. Forexample, when the elimination mechanism includes a spray nozzle with alinear spray pattern that is moved along the crop row perpendicular tothe longitudinal crop row axis, then plants growing substantiallyperpendicular to the longitudinal axis cannot be separately retained.The cost or utility function can be optimized for any other suitablevariable that influences the plant production. The aforementionedvariables can additionally be weighted within the cost or utilityfunction, such that the function can be tailored to favor optimizationof one variable over another. The weighting is preferably determinedautomatically (e.g. based on historical plant data), but canalternatively be determined or derived from a user input (e.g. the userindicates that plant yield is more important than plant size uniformity,so the absolute size of the plant is weighted higher than the plant sizeuniformity in the cost or utility function).

Removing plants from the plant field section S300 functions to generatethe selected retained plant pattern within the plant field sectioncaptured by the image. More preferably, crops are removed from the croprow section that was previously imaged, but any other suitable plant canbe removed from any other suitable field of plants. Plants correspondingto the unselected sub-regions are preferably removed, but plantscorresponding to the selected sub-regions can alternatively be removed.Removing plants from the field preferably includes generating plantremoval instructions, sending the plant removal instructions to theelimination mechanism, and subsequently removing the plant with theelimination mechanism.

Generating plant removal instructions preferably includes determining anoperation mode for each of a plurality of future time points, whereinthe elimination mechanism can be operated in plant removal mode or instandby mode. The elimination mechanism is preferably instructed tooperate in plant removal mode when a plant corresponding to anunselected sub-region is encountered, and is preferably instructed tooperate in standby mode when a plant corresponding to a selectedsub-region is encountered. However, the elimination mechanism can beoperated in any other suitable manner. The future time points arepreferably the times at which the elimination mechanism will be locatedproximal or adjacent the respective plant (e.g. when the eliminationmechanism position will substantially coincide with the position of therespective plant). The future time points are preferably determined bydetermining the velocity of the system at the current time point,estimating the time duration required for the elimination mechanism tobe adjacent the respective plant based on the velocity, the distancebetween the detector and the elimination mechanism, and the position ofthe respective plant on the virtual map (e.g. a position of theunselected region within the first image at the current time point),wherein the position of the system (e.g. the position of the detector)is preferably tracked on the virtual map, and selecting the future timepoint, wherein the duration between the future time point and thecurrent time point is preferably substantially equivalent to theestimated time duration. In one variation of the method, the eliminationmechanism is instructed to operate in plant removal mode by default, andis instructed to operate in standby mode a predetermined time prior tothe estimated time of encounter with plant to be retained. Theelimination mechanism is preferably operated in the standby mode untilthe elimination mechanism has passed the plant to be retained. Theduration of standby mode operation is preferably determined from theacceleration of the crop thinning system and the dimensions of thesub-region associated with said plant. Generating the plant removalinstructions can alternatively include determining an operation mode foreach of a plurality of geographic locations, wherein each geographiclocation preferably corresponds with a plant, as determined from thevirtual map. Generating the plant removal instructions can additionallyinclude tracking the plant to be removed with the computer vision system(e.g., through successive images) until the plant is estimated to belocated proximal the elimination mechanism.

However, any other suitable plant removal instructions can be generated.

Generating the plant removal instructions can additionally includeselecting a elimination mechanism. This step is preferably performedwhen the crop thinning system includes more than one eliminationmechanism. In one variation, the crop thinning system includes multipleelimination mechanisms of the same type, wherein generating plantremoval instructions includes selecting the elimination mechanismproximal the plant to be removed or selecting the elimination mechanismhaving a trajectory capable of necrosing all or a subset of the plantsto be removed. For example, the crop thinning system can include twonozzles fluidly connected to a removal fluid, wherein the two nozzlesare configured to travel on each side of the space defined between rows,such that a first nozzle is proximal the first row and a second nozzleis proximal the second row. The crop thinning system simultaneouslyimages sections of two rows, and generates instructions that includewhen to operate each nozzle (e.g. wherein the first nozzle is instructedto operate when a plant within the first row is to be removed, etc.). Inanother variation, the crop thinning system includes multiple types ofelimination mechanisms, wherein generating plant removal instructionsincludes selecting the appropriate elimination mechanism based on theproperties of the plant to be removed and the properties of neighboringplants. For example, generating plant removal instructions can includeselecting a spray nozzle when plants should be removed from an areasubstantially equivalent or larger than the width of the nozzle sprayand selecting a knife when the plant to be removed is surrounded byplants to be retained.

The plant removal instructions are preferably sent by the processor tothe elimination mechanism before the elimination mechanism encountersthe plant to be removed or retained. The plant removal instructions arepreferably sent prior to the estimated future time point, but canalternatively be sent to the elimination mechanism prior to the cropthinning system travelling a distance equivalent to the distance betweenthe detection mechanism and the elimination mechanism.

Removing the plants with the elimination mechanism preferably includesoperating the elimination mechanism in the plant removal mode at theinstructed time point or location. Removing the plants with theelimination mechanism can additionally include operating the eliminationmechanism in the standby mode at the respective instructed time point orlocation. Operating the crop thinning system in plant removal mode caninclude spraying a removal fluid at a predetermined concentration,operating a cutting mechanism (e.g. a hoe or a scythe), operating anuprooting mechanism, generating directional heat, generating directionalelectricity, or include any other suitable means of facilitating plantnecrosis. Operating the crop thinning system in standby mode can includespraying the removal fluid at a second predetermined concentration,spraying the removal fluid at a second flow rate lower than that of theplant removal mode, halting removal fluid provision to the nozzle,blocking the removal fluid spray, halting cutting mechanism operation,reducing the amount of generated heat, or include any other suitablemeans of retaining plants.

2. The Crop Thinning System

As shown in FIG. 8, the system for automated crop thinning 100 includesa detection mechanism 200 and an elimination mechanism 300. The system100 functions to image sections of a field of plants, more preferablysections of a crop row 10, determine whether to remove or retain eachplant within each imaged segment, and remove the plants marked forremoval as the system 100 moves along the crop row 10. More preferably,the system 100 functions to simultaneously image a section of multiplecrop rows 10 (e.g. adjacent crop rows 10), determine whether to removeor retain each plant within the imaged sections, and remove the plantsmarked for removal from the respective rows. The system 100 preferablyincludes multiple detection mechanisms 200, each configured to image asingle crop row 10, but can alternatively include a single detectionmechanism 200 that images a section of multiple crop rows 10 (e.g. thefield of view of the detection mechanism 200 spans multiple row widths),or a single detection mechanism 200 that images a section of a singlecrop row 10. The system 100 can include a single elimination mechanism300 that receives plant removal instructions based on the images fromthe one or more detection mechanisms 200, or can include multipleelimination mechanisms 300, wherein each elimination mechanism 300receives instructions from a single detection mechanism 200. The croprows 10 are preferably crop rows, but can alternatively be rows of weedsor any other suitable crop row.

The detection mechanism 200 is preferably coupled to the system 100 aknown distance away from the elimination mechanism 300, wherein thedetection mechanism 200 is preferably statically coupled (e.g. fixed)but can alternatively be movably coupled (e.g. with an adjustablebracket) to the elimination mechanism 300. The system 100 is preferablyconfigured to transiently couple (e.g. removably couple) to a drivemechanism, such as a tractor, through a coupling mechanism 400. However,the system 100 can alternatively include a drive mechanism that movesthe system 100 along the crop row 10. The system 100 is preferablyconfigured such that the detection mechanism 200 encounters a crop rowsection before the elimination mechanism 300 encounters the crop rowsection. In one variation, the detection mechanism 200 is mountedbetween a tractor couple (e.g. hitch couple) and the eliminationmechanism 300. The elimination mechanism 300 and detection mechanism 200are preferably arranged such that the centerlines of the detectionmechanism 200 (e.g. centerline of the field of view) and the eliminationmechanism 300 are aligned, but can alternatively be arranged such thatthe centerlines are offset. In the latter variation, the amount ofoffset is preferably known and accounted for during plant removalinstruction generation. The system 100 is preferably configured totraverse along each crop row 10 once, but can alternatively traversealong each crop row 10 multiple times. The system 100 preferablytraverses along a path substantially parallel to the longitudinal axisof the crop row 10, but can alternatively traverse along a pathperpendicular to the longitudinal axis of the crop rows 10, zigzagacross the width of the crop row 10, or travel any suitable path alongthe crop row 10. The system 100 is preferably configured to traverseover the top of the crop row 10 (e.g. a normal vector extending from theground 12 preferably intersects a portion of the system 100, morepreferably the detection mechanism 200 and/or elimination mechanism300), but can alternatively traverse along the side of the crop row 10,below the crop row 10, or in any suitable position relative to the croprow 10. The detection mechanism 200 and elimination mechanism 300position above the ground 12 is preferably defined and maintained by atransportation mechanism on the system 100 (e.g. the wheel radius, theheight of a bracket above ground 12), but can alternatively beadjustable (e.g. maintained by an adjustable bracket).

The detection mechanism 200 functions to image the plants within thecrop row 10. The detection mechanism 200 preferably includes an emitter220 and a detector 240, wherein the emitter 220 directs radiation 222toward the plants and the detector 240 receives the radiation reflectedoff the plants 224. The emitter 220 and detector 240 are preferablyarranged in fixed relation on a bracket, but can alternatively bemovably coupled on the bracket.

The emitter 220 preferably includes a plurality of radiation emittingmechanisms, but can alternatively include a single radiation emittingmechanism. The emitter 220 is preferably substantially planar (e.g. theradiation emitting mechanisms are arranged in a plane), but canalternatively be circular, toroidal, semi-spherical, or have any othersuitable configuration. The radiation emitting mechanism preferablyemits electromagnetic signals (e.g. waves, waveforms, etc.), morepreferably visible light, most preferably visible light having awavelength between 450 nm-550 nm (e.g. blue, cyan, and/or green).Alternatively, the electromagnetic radiation emitting mechanism can emitvisible light, infrared, ultraviolet, microwave, or any other suitableelectromagnetic radiation. The electromagnetic radiation emittingmechanism is preferably a light, but can alternatively be ahyperspectral emitter 220, a laser, an x-ray emitter 220, or any othersuitable electromagnetic emitter 220. Alternatively, the radiationemitting mechanism can emit acoustic signals, wherein the radiationemitting mechanism can emit ultrasound, infrasound, audible sound, orany other suitable waveform. The frequency of emitted radiation from theemitter 220 is preferably static (e.g. cannot be changed), but canalternatively be adjustable. The emitter intensity is preferablyadjustable between a maximum intensity (e.g. maximum applied power) anda minimum intensity (e.g. no applied power), but can alternatively beoperable in only one of an on state (e.g. applied power) or an off state(e.g. no applied power). The multiple emitters 200 can be controlledindependently or as a set. The frequency, intensity, and/or any othersuitable radiation parameter can be adjusted in response to a change inthe imaging conditions (e.g. wherein more visible light needs to beprovided to meet a brightness threshold in response to a decrease invisible light provided by the sun). In one variation, the emitter 220includes a plurality of light emitting diodes arranged in a planarconfiguration, wherein the light emitting diodes include white lightemitting diodes and cyan light emitting diodes. The white light emittingdiodes are preferably arranged in rows alternating with rows of cyanlight emitting diodes, but the cyan light emitting diodes canalternatively be arranged within the same row as the white lightemitting diodes. The ratio of the white light emitting diodes to cyanlight emitting diodes is preferably 1:1, but can alternatively be higheror lower.

The detector 240 is preferably a camera that records the electromagneticradiation reflected off a plant, but can alternatively include atransducer, an IR sensor, UV sensor, or any other suitable detector 240.The detector 240 is preferably a digital camera, but can alternativelybe an analog camera. The detector 240 is can include a charge coupleddevice (CCD) or a CMOS sensor that records the images. The detector 240preferably has a fixed field of view (FOV) or angle of view, but canalternatively have an adjustable field of view. In the latter variation,the field of view can be adjusted based on the distance of the detector240 from the plant or based on any other suitable imaging parameter. Thedetector 240 preferably generates frames of pixel data, but canalternatively generate waveforms or any other suitable signal

The detection mechanism 200 is preferably configured such that theemitter 220 casts a shadow at the plant center 20 and illuminates theplant edges 40 (as shown in FIG. 11). To cast a shadow at the plantcenter, the detection mechanism 200 preferably directs radiation at theplant at a low incidence angle. As shown in FIGS. 9 and 10, theradiation is preferably directed at the plant at an angle between theground 12 (e.g. 90 degrees) and a normal vector to the ground (e.g. 0degrees), more preferably at an angle between the normal vector and theviewing plane (e.g. 90 degrees from vertical), even more preferably atan angle between 30 degrees and 60 degrees from the normal vector, suchas at 45 degrees. The detection mechanism 200 is preferably configuredsuch that the detector 240 records the light reflected off the plantwithout blocking plant radiation. The detector 240 is preferablyarranged above the crop row 10, such that the detector 240 receivesradiation reflected in a vector substantially parallel to the normalvector to the ground 12 or parallel to a gravity vector. The detector240 is preferably arranged with the detector view plane substantiallyparallel to the ground 12 beneath the crop row section, but canalternatively be arranged with the detector view plane at an angle tothe ground 12. In one variation, the detection mechanism 200 preferablyincludes two substantially planar emitters 220 configured to directradiation from above the crop row 10 at an angle to the plant, with thedetector 240 arranged between the emitters 220. The distance between theemitters 220 is preferably substantially equivalent to or wider than astandard crop row width, but can alternatively be narrower than thestandard crop row width. The detection mechanism 200 preferably retainsthe detector 240 and emitter 220 a predetermined distance from the croprow section to be imaged, but can alternatively retain the detector 240and emitter 220 at a predetermined distance from the ground 12 adjacentthe crop row 10. In another variation, the detection mechanism 200includes an emitter 220 including a plurality of radiation emittingmechanisms arranged in a ring, with the detector 240 arrangedsubstantially coaxially with the ring. In another variation, thedetection mechanism 200 includes an emitter 220 directed toward amirror, wherein the mirror directs radiation toward the plant. Inanother variation, the detection mechanism 200 includes a mirror thatreflects the radiation reflected off the plants toward the camera. Inanother variation, the detection mechanism 200 includes a diffuser todiffuse the radiation applied to the plant and ground. In anothervariation, the detection mechanism 200 includes a lens that selectivelyfocuses or de-focuses the radiation on the plant and the ground.However, the detection mechanism 200 can be configured in any othersuitable manner.

The detection mechanism 200 can additionally function to controlenvironmental influence on the imaging environment. In one variation,the detection mechanism 200 can substantially block radiation havingsimilar wavelengths to those emitted by the emitter 220 from reachingthe crop row 10 segment during imaging. In one example, the detectionmechanism 200 is arranged such that the emitter 220, detector 240,bracket, and elimination mechanism 300 cooperatively substantially blocksunlight from the section to be imaged. However, the detection mechanism200 can control the imaging environment in any other suitable manner.

The elimination mechanism 300 functions to facilitate plant necrosis,but can alternatively physically remove the plant from the field orotherwise eliminate the plant. The crop thinning system 100 preferablyincludes one or more nozzles fluidly connected to a reservoir of removalfluid, wherein application of the removal fluid to the plant at a givenconcentration initializes plant necrosis. The removal fluid ispreferably fertilizer, wherein application of the fertilizer at highconcentrations (e.g. 100 gallons per acre of AN20 fertilizer) necrosesthe plant. However, the removal fluid can be water (e.g. applied at highpressure or high temperature), steam, herbicide, pesticide, bleach, orany other suitable fluid that facilitates the necrosis or physicalremoval of the plant from its position within the crop row 10. The cropthinning system 100 can alternatively include one or more blades, aheating device (e.g. an infrared applicator, microwave applicator,etc.), a radiation device (e.g. device that emits ultrasound atplant-necrosing wavelengths), or any other suitable mechanism thateradicates or facilitates eradication of a plant.

In one variation of the system 100, the system 100 includes multiplespray nozzles arranged in a linear pattern and configured to travelperpendicular to the longitudinal axis of the crop rows 60. Each spraynozzle is preferably configured to remove plants from an adjacent row.In one particular variation, as shown in FIG. 10, the system 100includes two spray nozzles, wherein the spray nozzles are preferablyfixed to the system 100 and travels within the inter-row space, belowthe visual plane of the detector 240 (e.g. below the tops of theplants), such that a first nozzle travels adjacent a first row, and asecond nozzle travels adjacent a second row. In operation, the system100 preferably selects a nozzle for use based on the row in which theplant to be removed resides (e.g. the first nozzle is selected if theplant is in the first row). The spray nozzles preferably create a widefan spray pattern, but can alternatively create a narrow fan spraypattern (e.g., a flat spray pattern), a cone spray pattern, a pointspray pattern, or any other suitable spray pattern. The spray nozzlescan be plain-orifice nozzles, shaped-orifice nozzles,surface-impingement nozzles, pressure-swirl nozzles, solid cone nozzles,compound nozzles, or any other suitable nozzle. The spray nozzles arepreferably single fluid nozzles, but can alternatively be two-fluidnozzles, wherein the two fluids preferably react to produce aplant-necrosing compound. In another variation, the system 100 includestwo nozzles in a linear arrangement, wherein the arrangement isconfigured to travel along an axis parallel to the crop row longitudinalaxis. The nozzle proximal the detection mechanism 200 is preferablyfluidly connected to a first removal fluid and the nozzle distal thedetection mechanism 200 is preferably fluidly connected to a secondremoval fluid, wherein the reaction product of the first and secondremoval fluids preferably necroses the plant (e.g. suffocates the plant,chemically burns the plant, forms a pesticide, etc.). The first and/orsecond removal fluid can independently function as a fertilizer for theplant. In operation, either the first or the second removal fluid can becontinually applied to the crop row 10, while the remaining removalfluid can be applied to the crop row 10 when a plant to be removed isencountered. In another variation, the system 100 includes three spraynozzles fixed to the system 100, wherein the spray nozzles arepreferably configured to travel above the crop rows 10, at or above thevisual plane of the detector 240 (e.g. above the tops of the plants).The spacing between each of the nozzles is preferably substantiallyequivalent to the spacing between the longitudinal axes of the adjacentcrop rows 10, such that the first nozzle is substantially aligned withthe first row, the second nozzle is substantially aligned with thesecond row, and the third nozzle is substantially aligned with the thirdrow. In operation, the system 100 preferably selects a nozzle for usebased on the row in which the plant to be removed resides (e.g. thefirst nozzle is selected if the plant is in the first row). In anothervariation, the system 100 can include one or more nozzles that actuates,rotating about a rotational axis to aim the removal fluid spray at theplant to be removed. In another variation, the system 100 can includeone or more nozzles with fixed angular positions that are moveablycoupled to a bracket, such that the linear position of each nozzle canbe adjusted. In operation, the system 100 moves a nozzle above the areain which a plant is estimated to be located.

The system 100 can additionally include a processor that functions toselect plants for retention, or conversely, select plants for removal.The processor can additionally function to process the image receivedfrom the detection mechanism 200 to identify individual, contiguousplants. The processor can additionally function to generate removalinstructions for the plants selected for removal. The processor canadditionally include memory, wherein the memory can store retained plantinformation (e.g. position, size, shape, etc.), removed plantinformation, user preferences, target plant parameters (e.g. targetyield, target plant density, target size, target uniformity, etc.), orany other suitable information.

The systems and methods of the preferred embodiment can be embodiedand/or implemented at least in part as a machine configured to receive acomputer-readable medium storing computer-readable instructions. Theinstructions are preferably executed by computer-executable componentspreferably integrated within a towable vehicle plant-thinning vehicle, amobile plant-thinning vehicle, an autonomous plant-thinning vehicle, orany other suitable machine or vehicle. Other systems 100 and methods ofthe preferred embodiment can be embodied and/or implemented at least inpart as a machine configured to receive a computer-readable mediumstoring computer-readable instructions. The instructions are preferablyexecuted by computer-executable components preferably integrated withapparatuses and networks of the type described above. Thecomputer-readable medium can be stored on any suitable computer readablemedia such as RAMs, ROMs, flash memory, EEPROMs, optical devices (CD orDVD), hard drives, floppy drives, or any suitable device. Thecomputer-executable component is preferably a processor but any suitablededicated hardware device can (alternatively or additionally) executethe instructions.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the preferred embodiments of the invention withoutdeparting from the scope of this invention as defined in the followingclaims.

What is claimed is:
 1. A method comprising: accessing an image of afield comprising a plurality of plants, the image captured by a farmingmachine moving through the field on a current pass; segmenting the imageinto a plurality of field regions, each of the field regions localizinga plant of the plurality of plants within a two dimensional area in theimage; selecting, for each field region of the plurality of fieldregions, a spray treatment for the plant localized in the field regionbased on characteristics describing the localized plant and parametersdescribing a plurality of prior plants treated by the farming machine ona previous pass of the farming machine through the field; and sendingtreatment instructions for each region of the plurality of field regionsto a plurality of spray mechanisms of the farming machine, the treatmentinstructions indicating the spray treatment to apply to each regionbefore the farming machine travels past the region in the field on thecurrent pass.